
# без этого не работает ydata_profiling
!pip install --upgrade pillow

Requirement already satisfied: pillow in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (10.0.0)

WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)


!pip install ydata_profiling -U

Requirement already satisfied: ydata_profiling in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (4.5.1)
Requirement already satisfied: dacite>=1.8 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.8.1)
Requirement already satisfied: wordcloud>=1.9.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.9.2)
Requirement already satisfied: statsmodels<1,>=0.13.2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (0.13.2)
Requirement already satisfied: pandas!=1.4.0,<2.1,>1.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.2.4)
Requirement already satisfied: requests<3,>=2.24.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (2.28.1)
Requirement already satisfied: scipy<1.12,>=1.4.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.8.0)
Requirement already satisfied: imagehash==4.3.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (4.3.1)
Requirement already satisfied: multimethod<2,>=1.4 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.9.1)
Requirement already satisfied: numpy<1.24,>=1.16.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.20.1)
Requirement already satisfied: typeguard<3,>=2.13.2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (2.13.3)
Requirement already satisfied: tqdm<5,>=4.48.2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (4.64.1)
Requirement already satisfied: visions[type_image_path]==0.7.5 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (0.7.5)
Requirement already satisfied: jinja2<3.2,>=2.11.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (3.1.2)
Requirement already satisfied: PyYAML<6.1,>=5.0.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (6.0)
Requirement already satisfied: htmlmin==0.1.12 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (0.1.12)
Requirement already satisfied: phik<0.13,>=0.11.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (0.12.3)
Requirement already satisfied: pydantic<2,>=1.8.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (1.10.12)
Requirement already satisfied: seaborn<0.13,>=0.10.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (0.11.1)
Requirement already satisfied: matplotlib<4,>=3.2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from ydata_profiling) (3.3.4)
Requirement already satisfied: PyWavelets in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from imagehash==4.3.1->ydata_profiling) (1.4.1)
Requirement already satisfied: pillow in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from imagehash==4.3.1->ydata_profiling) (10.0.0)
Requirement already satisfied: networkx>=2.4 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from visions[type_image_path]==0.7.5->ydata_profiling) (3.1)
Requirement already satisfied: tangled-up-in-unicode>=0.0.4 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from visions[type_image_path]==0.7.5->ydata_profiling) (0.2.0)
Requirement already satisfied: attrs>=19.3.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from visions[type_image_path]==0.7.5->ydata_profiling) (22.2.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from jinja2<3.2,>=2.11.1->ydata_profiling) (2.1.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from matplotlib<4,>=3.2->ydata_profiling) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from matplotlib<4,>=3.2->ydata_profiling) (1.4.4)
Requirement already satisfied: cycler>=0.10 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from matplotlib<4,>=3.2->ydata_profiling) (0.11.0)
Requirement already satisfied: python-dateutil>=2.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from matplotlib<4,>=3.2->ydata_profiling) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from pandas!=1.4.0,<2.1,>1.1->ydata_profiling) (2022.7)
Requirement already satisfied: joblib>=0.14.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from phik<0.13,>=0.11.1->ydata_profiling) (1.2.0)
Requirement already satisfied: typing-extensions>=4.2.0 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from pydantic<2,>=1.8.1->ydata_profiling) (4.4.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from requests<3,>=2.24.0->ydata_profiling) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from requests<3,>=2.24.0->ydata_profiling) (2022.12.7)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from requests<3,>=2.24.0->ydata_profiling) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from requests<3,>=2.24.0->ydata_profiling) (3.4)
Requirement already satisfied: patsy>=0.5.2 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from statsmodels<1,>=0.13.2->ydata_profiling) (0.5.3)
Requirement already satisfied: packaging>=21.3 in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from statsmodels<1,>=0.13.2->ydata_profiling) (22.0)
Requirement already satisfied: colorama in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from tqdm<5,>=4.48.2->ydata_profiling) (0.4.6)
Requirement already satisfied: six in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from patsy>=0.5.2->statsmodels<1,>=0.13.2->ydata_profiling) (1.16.0)

WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)


pip install xgboost

Requirement already satisfied: xgboost in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (1.7.6)
Requirement already satisfied: numpy in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from xgboost) (1.20.1)
Requirement already satisfied: scipy in c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages (from xgboost) (1.8.0)
Note: you may need to restart the kernel to use updated packages.

WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\alexey_tomashevskiy\anaconda3\envs\practicum_sprint_22\lib\site-packages)


import pandas as pd
import numpy as np

# для анализа данных
from ydata_profiling import ProfileReport

# для графиков
import matplotlib.pyplot as plt
import seaborn as sns

# для скрытия ошибок
import warnings
# для корректного отражения данных
from IPython.display import display
#для загрузки данных и с сервера и локально
import os

# регулярные выражения
import re

# для разботы с разницами времени
from datetime import datetime

# для разбиение на выборки
from sklearn.model_selection import train_test_split
# метрика MSE
from sklearn.metrics import mean_squared_error
# метрика MAE
from sklearn.metrics import mean_absolute_error
# для подбора параметров моделей
from sklearn.model_selection import GridSearchCV, KFold
# регрессор для проверки адекватности
from sklearn.dummy import DummyRegressor
# # для раздельной предобработки количествнных и категорийных признаков
# from sklearn.compose import ColumnTransformer
# Линейная регрессия
from sklearn.linear_model import LinearRegression
# Ridge регрессия
from sklearn.linear_model import Ridge
# Lasso регрессия
from sklearn.linear_model import Lasso
# ElasticNet регрессия
from sklearn.linear_model import ElasticNet
# регрессор Дерево Решений
from sklearn.tree import DecisionTreeRegressor
# регрессор Случайный Лес
from sklearn.ensemble import RandomForestRegressor
# для создания конвеера/ трудопровода
from sklearn.pipeline import Pipeline, make_pipeline
# для маштабирования признаков
from sklearn.preprocessing import StandardScaler
# заполнение пропусков
from sklearn.impute import SimpleImputer


# градиентный бустинг XGBoost
import xgboost as xgb


# для скрытия ошибок
warnings.filterwarnings("ignore")

# для увеличения окна вывода 
pd.options.display.max_rows = 300
# максимальная вместимость ячейки для отображения
pd.set_option('display.max_colwidth', None)
# четыре знака после запятой
pd.set_option('display.float_format', '{:.4f}'.format)

# для задания размера графиков по умолчанию
plt.rcParams["figure.figsize"] = (7, 5)

# для псевдорандомных значений
np.random.seed(seed=15823)
np.random.RandomState(seed=15823)

RandomState(MT19937) at 0x1EF8CD17E40


def df_load_func(file_name):
    #путь с сервероной версии
    pth1 = '/datasets/' + file_name
    # моя локальная версия
    pth2 = file_name

    # загрузка данных
    if os.path.exists(pth1):
        df_name = pd.read_csv(pth1)
    elif os.path.exists(pth2):
        df_name = pd.read_csv(pth2)
    else:
        display('Проверьте пути к файлам с данными!')

    display(df_name.info())    
    display(df_name.head())
    return df_name


df_electrod = df_load_func('data_arc_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 14876 entries, 0 to 14875
Data columns (total 5 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   key                   14876 non-null  int64  
 1   Начало нагрева дугой  14876 non-null  object 
 2   Конец нагрева дугой   14876 non-null  object 
 3   Активная мощность     14876 non-null  float64
 4   Реактивная мощность   14876 non-null  float64
dtypes: float64(2), int64(1), object(2)
memory usage: 581.2+ KB

None


df_bulk_additives_volume = df_load_func('data_bulk_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3129 entries, 0 to 3128
Data columns (total 16 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   key      3129 non-null   int64  
 1   Bulk 1   252 non-null    float64
 2   Bulk 2   22 non-null     float64
 3   Bulk 3   1298 non-null   float64
 4   Bulk 4   1014 non-null   float64
 5   Bulk 5   77 non-null     float64
 6   Bulk 6   576 non-null    float64
 7   Bulk 7   25 non-null     float64
 8   Bulk 8   1 non-null      float64
 9   Bulk 9   19 non-null     float64
 10  Bulk 10  176 non-null    float64
 11  Bulk 11  177 non-null    float64
 12  Bulk 12  2450 non-null   float64
 13  Bulk 13  18 non-null     float64
 14  Bulk 14  2806 non-null   float64
 15  Bulk 15  2248 non-null   float64
dtypes: float64(15), int64(1)
memory usage: 391.2 KB

None


df_bulk_additives_time = df_load_func('data_bulk_time_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3129 entries, 0 to 3128
Data columns (total 16 columns):
 #   Column   Non-Null Count  Dtype 
---  ------   --------------  ----- 
 0   key      3129 non-null   int64 
 1   Bulk 1   252 non-null    object
 2   Bulk 2   22 non-null     object
 3   Bulk 3   1298 non-null   object
 4   Bulk 4   1014 non-null   object
 5   Bulk 5   77 non-null     object
 6   Bulk 6   576 non-null    object
 7   Bulk 7   25 non-null     object
 8   Bulk 8   1 non-null      object
 9   Bulk 9   19 non-null     object
 10  Bulk 10  176 non-null    object
 11  Bulk 11  177 non-null    object
 12  Bulk 12  2450 non-null   object
 13  Bulk 13  18 non-null     object
 14  Bulk 14  2806 non-null   object
 15  Bulk 15  2248 non-null   object
dtypes: int64(1), object(15)
memory usage: 391.2+ KB

None


df_gaz_info = df_load_func('data_gas_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3239 entries, 0 to 3238
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   key     3239 non-null   int64  
 1   Газ 1   3239 non-null   float64
dtypes: float64(1), int64(1)
memory usage: 50.7 KB

None


df_temp_info = df_load_func('data_temp_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 18092 entries, 0 to 18091
Data columns (total 3 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   key           18092 non-null  int64  
 1   Время замера  18092 non-null  object 
 2   Температура   14665 non-null  float64
dtypes: float64(1), int64(1), object(1)
memory usage: 424.2+ KB

None


df_wire_additives_volume = df_load_func('data_wire_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3081 entries, 0 to 3080
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   key     3081 non-null   int64  
 1   Wire 1  3055 non-null   float64
 2   Wire 2  1079 non-null   float64
 3   Wire 3  63 non-null     float64
 4   Wire 4  14 non-null     float64
 5   Wire 5  1 non-null      float64
 6   Wire 6  73 non-null     float64
 7   Wire 7  11 non-null     float64
 8   Wire 8  19 non-null     float64
 9   Wire 9  29 non-null     float64
dtypes: float64(9), int64(1)
memory usage: 240.8 KB

None


df_wire_additives_time = df_load_func('data_wire_time_new.csv')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3081 entries, 0 to 3080
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   key     3081 non-null   int64 
 1   Wire 1  3055 non-null   object
 2   Wire 2  1079 non-null   object
 3   Wire 3  63 non-null     object
 4   Wire 4  14 non-null     object
 5   Wire 5  1 non-null      object
 6   Wire 6  73 non-null     object
 7   Wire 7  11 non-null     object
 8   Wire 8  19 non-null     object
 9   Wire 9  29 non-null     object
dtypes: int64(1), object(9)
memory usage: 240.8+ KB

None


def columns_rename(df):
    return ['_'.join(re.sub(r'([A-Z])', r' \1', col).split()).lower() for col in df.columns]


df_electrod.columns

Index(['key', 'Начало нагрева дугой', 'Конец нагрева дугой',
       'Активная мощность', 'Реактивная мощность'],
      dtype='object')


df_electrod.columns = ['key', 'time_start_heating', 'time_end_heating', 'active_power', 'reactive_power']
df_electrod.columns

Index(['key', 'time_start_heating', 'time_end_heating', 'active_power',
       'reactive_power'],
      dtype='object')


df_bulk_additives_volume.columns

Index(['key', 'Bulk 1', 'Bulk 2', 'Bulk 3', 'Bulk 4', 'Bulk 5', 'Bulk 6',
       'Bulk 7', 'Bulk 8', 'Bulk 9', 'Bulk 10', 'Bulk 11', 'Bulk 12',
       'Bulk 13', 'Bulk 14', 'Bulk 15'],
      dtype='object')


df_bulk_additives_volume.columns = columns_rename(df_bulk_additives_volume)
df_bulk_additives_volume.columns

Index(['key', 'bulk_1', 'bulk_2', 'bulk_3', 'bulk_4', 'bulk_5', 'bulk_6',
       'bulk_7', 'bulk_8', 'bulk_9', 'bulk_10', 'bulk_11', 'bulk_12',
       'bulk_13', 'bulk_14', 'bulk_15'],
      dtype='object')


df_bulk_additives_time.columns

Index(['key', 'Bulk 1', 'Bulk 2', 'Bulk 3', 'Bulk 4', 'Bulk 5', 'Bulk 6',
       'Bulk 7', 'Bulk 8', 'Bulk 9', 'Bulk 10', 'Bulk 11', 'Bulk 12',
       'Bulk 13', 'Bulk 14', 'Bulk 15'],
      dtype='object')


df_bulk_additives_time.columns = columns_rename(df_bulk_additives_time)
df_bulk_additives_time.columns

Index(['key', 'bulk_1', 'bulk_2', 'bulk_3', 'bulk_4', 'bulk_5', 'bulk_6',
       'bulk_7', 'bulk_8', 'bulk_9', 'bulk_10', 'bulk_11', 'bulk_12',
       'bulk_13', 'bulk_14', 'bulk_15'],
      dtype='object')


df_gaz_info.columns

Index(['key', 'Газ 1'], dtype='object')


df_gaz_info.columns = ['key', 'gaz']
df_gaz_info.columns

Index(['key', 'gaz'], dtype='object')


df_temp_info.columns

Index(['key', 'Время замера', 'Температура'], dtype='object')


df_temp_info.columns = ['key', 'time_temperature_test', 'temperature']
df_temp_info.columns

Index(['key', 'time_temperature_test', 'temperature'], dtype='object')


df_wire_additives_volume.columns

Index(['key', 'Wire 1', 'Wire 2', 'Wire 3', 'Wire 4', 'Wire 5', 'Wire 6',
       'Wire 7', 'Wire 8', 'Wire 9'],
      dtype='object')


df_wire_additives_volume.columns = columns_rename(df_wire_additives_volume)
df_wire_additives_volume.columns

Index(['key', 'wire_1', 'wire_2', 'wire_3', 'wire_4', 'wire_5', 'wire_6',
       'wire_7', 'wire_8', 'wire_9'],
      dtype='object')


df_wire_additives_time.columns

Index(['key', 'Wire 1', 'Wire 2', 'Wire 3', 'Wire 4', 'Wire 5', 'Wire 6',
       'Wire 7', 'Wire 8', 'Wire 9'],
      dtype='object')


df_wire_additives_time.columns = columns_rename(df_wire_additives_time)
df_wire_additives_time.columns

Index(['key', 'wire_1', 'wire_2', 'wire_3', 'wire_4', 'wire_5', 'wire_6',
       'wire_7', 'wire_8', 'wire_9'],
      dtype='object')


# Эта функция перебирает все столбцы и, если у данных столбца строчный тип, то меняет его на DateTime на настройках по умолчанию.
def from_object_to_datetime(df):
    for col in df.columns:
        if pd.api.types.is_string_dtype(df[col]):
                df[col]= pd.to_datetime(df[col])
    return df


df_electrod.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 14876 entries, 0 to 14875
Data columns (total 5 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   key                 14876 non-null  int64  
 1   time_start_heating  14876 non-null  object 
 2   time_end_heating    14876 non-null  object 
 3   active_power        14876 non-null  float64
 4   reactive_power      14876 non-null  float64
dtypes: float64(2), int64(1), object(2)
memory usage: 581.2+ KB


df_electrod = from_object_to_datetime(df_electrod)
df_electrod.info()
df_electrod.head()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 14876 entries, 0 to 14875
Data columns (total 5 columns):
 #   Column              Non-Null Count  Dtype         
---  ------              --------------  -----         
 0   key                 14876 non-null  int64         
 1   time_start_heating  14876 non-null  datetime64[ns]
 2   time_end_heating    14876 non-null  datetime64[ns]
 3   active_power        14876 non-null  float64       
 4   reactive_power      14876 non-null  float64       
dtypes: datetime64[ns](2), float64(2), int64(1)
memory usage: 581.2 KB


df_bulk_additives_time.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3129 entries, 0 to 3128
Data columns (total 16 columns):
 #   Column   Non-Null Count  Dtype 
---  ------   --------------  ----- 
 0   key      3129 non-null   int64 
 1   bulk_1   252 non-null    object
 2   bulk_2   22 non-null     object
 3   bulk_3   1298 non-null   object
 4   bulk_4   1014 non-null   object
 5   bulk_5   77 non-null     object
 6   bulk_6   576 non-null    object
 7   bulk_7   25 non-null     object
 8   bulk_8   1 non-null      object
 9   bulk_9   19 non-null     object
 10  bulk_10  176 non-null    object
 11  bulk_11  177 non-null    object
 12  bulk_12  2450 non-null   object
 13  bulk_13  18 non-null     object
 14  bulk_14  2806 non-null   object
 15  bulk_15  2248 non-null   object
dtypes: int64(1), object(15)
memory usage: 391.2+ KB


df_bulk_additives_time = from_object_to_datetime(df_bulk_additives_time)
df_bulk_additives_time.info()
df_bulk_additives_time.head()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3129 entries, 0 to 3128
Data columns (total 16 columns):
 #   Column   Non-Null Count  Dtype         
---  ------   --------------  -----         
 0   key      3129 non-null   int64         
 1   bulk_1   252 non-null    datetime64[ns]
 2   bulk_2   22 non-null     datetime64[ns]
 3   bulk_3   1298 non-null   datetime64[ns]
 4   bulk_4   1014 non-null   datetime64[ns]
 5   bulk_5   77 non-null     datetime64[ns]
 6   bulk_6   576 non-null    datetime64[ns]
 7   bulk_7   25 non-null     datetime64[ns]
 8   bulk_8   1 non-null      datetime64[ns]
 9   bulk_9   19 non-null     datetime64[ns]
 10  bulk_10  176 non-null    datetime64[ns]
 11  bulk_11  177 non-null    datetime64[ns]
 12  bulk_12  2450 non-null   datetime64[ns]
 13  bulk_13  18 non-null     datetime64[ns]
 14  bulk_14  2806 non-null   datetime64[ns]
 15  bulk_15  2248 non-null   datetime64[ns]
dtypes: datetime64[ns](15), int64(1)
memory usage: 391.2 KB


df_temp_info.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 18092 entries, 0 to 18091
Data columns (total 3 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   key                    18092 non-null  int64  
 1   time_temperature_test  18092 non-null  object 
 2   temperature            14665 non-null  float64
dtypes: float64(1), int64(1), object(1)
memory usage: 424.2+ KB


df_temp_info = from_object_to_datetime(df_temp_info)
df_temp_info.info()
df_temp_info.head()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 18092 entries, 0 to 18091
Data columns (total 3 columns):
 #   Column                 Non-Null Count  Dtype         
---  ------                 --------------  -----         
 0   key                    18092 non-null  int64         
 1   time_temperature_test  18092 non-null  datetime64[ns]
 2   temperature            14665 non-null  float64       
dtypes: datetime64[ns](1), float64(1), int64(1)
memory usage: 424.2 KB


df_wire_additives_time.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3081 entries, 0 to 3080
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   key     3081 non-null   int64 
 1   wire_1  3055 non-null   object
 2   wire_2  1079 non-null   object
 3   wire_3  63 non-null     object
 4   wire_4  14 non-null     object
 5   wire_5  1 non-null      object
 6   wire_6  73 non-null     object
 7   wire_7  11 non-null     object
 8   wire_8  19 non-null     object
 9   wire_9  29 non-null     object
dtypes: int64(1), object(9)
memory usage: 240.8+ KB


df_wire_additives_time = from_object_to_datetime(df_wire_additives_time)
df_wire_additives_time.info()
df_wire_additives_time.head()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3081 entries, 0 to 3080
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   key     3081 non-null   int64         
 1   wire_1  3055 non-null   datetime64[ns]
 2   wire_2  1079 non-null   datetime64[ns]
 3   wire_3  63 non-null     datetime64[ns]
 4   wire_4  14 non-null     datetime64[ns]
 5   wire_5  1 non-null      datetime64[ns]
 6   wire_6  73 non-null     datetime64[ns]
 7   wire_7  11 non-null     datetime64[ns]
 8   wire_8  19 non-null     datetime64[ns]
 9   wire_9  29 non-null     datetime64[ns]
dtypes: datetime64[ns](9), int64(1)
memory usage: 240.8 KB


profile_df_electrod = ProfileReport(df_electrod)
profile_df_electrod.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


df_electrod.loc[df_electrod['reactive_power'] < 0]


# создадим массив где указаны значение партий/key для которых реативная мощьность отрицательная
key_with_negative_reactive_power = df_electrod.query('reactive_power < 0')['key'].unique()
# удалим информацию о партиях, где раективная мощьность отрицательная
df_electrod.query('key not in @key_with_negative_reactive_power', inplace=True)


df_electrod.loc[df_electrod['reactive_power'] < 0]


df_electrod['reactive_power'].describe()

count   14872.0000
mean        0.4871
std         0.1976
min         0.1538
25%         0.3372
50%         0.4417
75%         0.6082
max         1.2703
Name: reactive_power, dtype: float64


df_electrod['active_power'].corr(df_electrod['reactive_power'])

0.9663821708188474


df_electrod['reactive_power'].hist(bins=50)
plt.title('Гистограмма значений Реактивной мощности(50 корзин)')
plt.xlabel('Значения, ВАр')
plt.ylabel('Количество значений, шт')
plt.show()


df_electrod.boxplot(['reactive_power'])
plt.title('График "с усами"')
plt.ylabel('Значения, ВАр')
plt.show()


profile_df_bulk_additives_volume = ProfileReport(df_bulk_additives_volume)
profile_df_bulk_additives_volume.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


df_bulk_additives_volume.fillna(0, inplace=True)
df_bulk_additives_volume.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3129 entries, 0 to 3128
Data columns (total 16 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   key      3129 non-null   int64  
 1   bulk_1   3129 non-null   float64
 2   bulk_2   3129 non-null   float64
 3   bulk_3   3129 non-null   float64
 4   bulk_4   3129 non-null   float64
 5   bulk_5   3129 non-null   float64
 6   bulk_6   3129 non-null   float64
 7   bulk_7   3129 non-null   float64
 8   bulk_8   3129 non-null   float64
 9   bulk_9   3129 non-null   float64
 10  bulk_10  3129 non-null   float64
 11  bulk_11  3129 non-null   float64
 12  bulk_12  3129 non-null   float64
 13  bulk_13  3129 non-null   float64
 14  bulk_14  3129 non-null   float64
 15  bulk_15  3129 non-null   float64
dtypes: float64(15), int64(1)
memory usage: 391.2 KB


profile_df_bulk_additives_time = ProfileReport(df_bulk_additives_time)
profile_df_bulk_additives_time.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


profile_df_gaz_info = ProfileReport(df_gaz_info)
profile_df_gaz_info.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


profile_df_temp_info = ProfileReport(df_temp_info)
profile_df_temp_info.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


key_with_1500_temp_less = list(df_temp_info.query('temperature < 1500')['key'].unique())
len(key_with_1500_temp_less)

5


key_with_2_temp_tests = (df_temp_info.groupby(['key'])['temperature'].count() >= 2)
key_with_2_temp_tests = list(key_with_2_temp_tests[key_with_2_temp_tests == True].index)
len(key_with_2_temp_tests)

2475


# нам нужно, чтобы был стартовый замер температуры и финальный

key_with_first_and_last_temp_tests = list((df_temp_info
                                           .pivot_table(index='key', values='temperature', aggfunc=['first', 'last'])
                                           .dropna() # удаляем значения, где первый или второй замер пропуски
                                          )
                                          .index)
len(key_with_first_and_last_temp_tests)

3216


df_temp_info.query('(key not in @key_with_1500_temp_less) & (key in @key_with_2_temp_tests) & (key in @key_with_first_and_last_temp_tests)'
                   , inplace=True)
df_temp_info['key'].unique().shape

(2471,)


profile_df_wire_additives_volume = ProfileReport(df_wire_additives_volume)
profile_df_wire_additives_volume.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


df_wire_additives_volume.fillna(0, inplace=True)
df_wire_additives_volume.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3081 entries, 0 to 3080
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   key     3081 non-null   int64  
 1   wire_1  3081 non-null   float64
 2   wire_2  3081 non-null   float64
 3   wire_3  3081 non-null   float64
 4   wire_4  3081 non-null   float64
 5   wire_5  3081 non-null   float64
 6   wire_6  3081 non-null   float64
 7   wire_7  3081 non-null   float64
 8   wire_8  3081 non-null   float64
 9   wire_9  3081 non-null   float64
dtypes: float64(9), int64(1)
memory usage: 240.8 KB


profile_df_wire_additives_time = ProfileReport(df_wire_additives_time)
profile_df_wire_additives_time.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


# найдем время когда идет нагрев
df_electrod['d_time'] = df_electrod['time_end_heating'] - df_electrod['time_start_heating']
# выразим его в секундах
df_electrod['d_time'] = df_electrod['d_time'].apply(lambda x: x.total_seconds())
# посчитаем количество энергии, которая идет на нагрев
df_electrod['heating_energy'] = df_electrod['d_time'] * df_electrod['active_power']
df_electrod.head()


df_electrod_pivot = pd.pivot_table(df_electrod, index='key'
                                   , values=['heating_energy', 'd_time', 'time_start_heating', 'time_end_heating']
                                   , aggfunc={'heating_energy': ['sum', 'count']
                                              , 'd_time': 'sum'
                                              , 'time_start_heating': 'min'
                                              , 'time_end_heating': 'max'
                                             },
                                  ).reset_index()

# найдем общее время от начального замера до последнего
df_electrod_pivot['total_time'] = df_electrod_pivot['time_end_heating', 'max'] - df_electrod_pivot['time_start_heating', 'min']
# выразим его в секундах
df_electrod_pivot['total_time'] = df_electrod_pivot['total_time'].apply(lambda x: x.total_seconds())

# удалим не нужные столбцы
df_electrod_pivot.drop(columns=[('time_end_heating', 'max')
                                , ('time_start_heating', 'min')
                               ]
                       , inplace=True
                      )

df_electrod_pivot.head()
df_electrod = df_electrod_pivot
df_electrod.head()


df_electrod.columns = ['key', 'heating_time', 'heating_cont', 'heating_energy'
                       , 'total_time'
                      ]
df_electrod.columns

Index(['key', 'heating_time', 'heating_cont', 'heating_energy', 'total_time'], dtype='object')


# сгруппируем данные по партиям, возьмем стратовое и финишное время
df_temp_info = (pd.pivot_table(df_temp_info, index='key', values='temperature', aggfunc=['first', 'last'])
                            .reset_index())
# дадим колонкам привычные названия
df_temp_info.columns = ['key', 'start_temp', 'finish_temp']
df_temp_info.head()


# первое объединение добавок с включением всех партий в обоих датафреймах
data = df_bulk_additives_volume.merge(df_wire_additives_volume
                                      , on='key'
#                                       , how='outer'
                                      , how='inner'
                                     )
display(data.shape)
# второе объединение с данными о энергии нагрева, здесь и далее объединение идёт по пересечению номеров партий
data = data.merge(df_electrod
                 , on='key'
                  , how='inner'
                 )
display(data.shape)
# третье объединение с данными о газе
data = data.merge(df_gaz_info
                 , on='key'
                  , how='inner'
                 )
display(data.shape)
# четвёртое объединение с данными о температурах
data = data.merge(df_temp_info
                 , on='key'
                  , how='inner'
                 )
display(data.shape)

(3025, 25)

(3023, 29)

(3021, 30)

(2324, 32)


data.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2324 entries, 0 to 2323
Data columns (total 32 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   key             2324 non-null   int64  
 1   bulk_1          2324 non-null   float64
 2   bulk_2          2324 non-null   float64
 3   bulk_3          2324 non-null   float64
 4   bulk_4          2324 non-null   float64
 5   bulk_5          2324 non-null   float64
 6   bulk_6          2324 non-null   float64
 7   bulk_7          2324 non-null   float64
 8   bulk_8          2324 non-null   float64
 9   bulk_9          2324 non-null   float64
 10  bulk_10         2324 non-null   float64
 11  bulk_11         2324 non-null   float64
 12  bulk_12         2324 non-null   float64
 13  bulk_13         2324 non-null   float64
 14  bulk_14         2324 non-null   float64
 15  bulk_15         2324 non-null   float64
 16  wire_1          2324 non-null   float64
 17  wire_2          2324 non-null   float64
 18  wire_3          2324 non-null   float64
 19  wire_4          2324 non-null   float64
 20  wire_5          2324 non-null   float64
 21  wire_6          2324 non-null   float64
 22  wire_7          2324 non-null   float64
 23  wire_8          2324 non-null   float64
 24  wire_9          2324 non-null   float64
 25  heating_time    2324 non-null   float64
 26  heating_cont    2324 non-null   int64  
 27  heating_energy  2324 non-null   float64
 28  total_time      2324 non-null   float64
 29  gaz             2324 non-null   float64
 30  start_temp      2324 non-null   float64
 31  finish_temp     2324 non-null   float64
dtypes: float64(30), int64(2)
memory usage: 599.2 KB


# включили минимальные отчет, так как при нормальном сильно замедляется работа. А подробно данные уже изучены.
profile_data = ProfileReport(data, minimal=True)
profile_data.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]

Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]


data.set_index('key', inplace=True)
data.head()


rs = 15823


target = data['finish_temp']
target.describe()

count   2324.0000
mean    1593.3726
std       11.2075
min     1541.0000
25%     1587.0000
50%     1593.0000
75%     1598.0000
max     1653.0000
Name: finish_temp, dtype: float64


features = data.drop(columns='finish_temp')
features.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2324 entries, 1 to 2499
Data columns (total 30 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   bulk_1          2324 non-null   float64
 1   bulk_2          2324 non-null   float64
 2   bulk_3          2324 non-null   float64
 3   bulk_4          2324 non-null   float64
 4   bulk_5          2324 non-null   float64
 5   bulk_6          2324 non-null   float64
 6   bulk_7          2324 non-null   float64
 7   bulk_8          2324 non-null   float64
 8   bulk_9          2324 non-null   float64
 9   bulk_10         2324 non-null   float64
 10  bulk_11         2324 non-null   float64
 11  bulk_12         2324 non-null   float64
 12  bulk_13         2324 non-null   float64
 13  bulk_14         2324 non-null   float64
 14  bulk_15         2324 non-null   float64
 15  wire_1          2324 non-null   float64
 16  wire_2          2324 non-null   float64
 17  wire_3          2324 non-null   float64
 18  wire_4          2324 non-null   float64
 19  wire_5          2324 non-null   float64
 20  wire_6          2324 non-null   float64
 21  wire_7          2324 non-null   float64
 22  wire_8          2324 non-null   float64
 23  wire_9          2324 non-null   float64
 24  heating_time    2324 non-null   float64
 25  heating_cont    2324 non-null   int64  
 26  heating_energy  2324 non-null   float64
 27  total_time      2324 non-null   float64
 28  gaz             2324 non-null   float64
 29  start_temp      2324 non-null   float64
dtypes: float64(29), int64(1)
memory usage: 562.8 KB


# 25% от 100%
features_train, features_test, target_train, target_test = train_test_split(features, target
                                                                            , test_size=0.25                                                                            
                                                                            , random_state=rs
                                                                            , shuffle=True
#                                                                             , stratify=target
                                                                           )
display(features.shape, features_train.shape, features_test.shape)
display(target.shape, target_train.shape, target_test.shape)

(2324, 30)

(1743, 30)

(581, 30)

(2324,)

(1743,)

(581,)


cv_folders = KFold(n_splits=5
                   , shuffle=True
                   , random_state=rs
                  )


dummy_model = DummyRegressor(strategy='median')
dummy_model.fit(features_train, target_train)
dummy_prediction = dummy_model.predict(features_train)
mean_absolute_error(target_train, dummy_prediction)

8.059093516924841


dummy_model = DummyRegressor(strategy='median')
dummy_model.fit(features_train, target_train)
dummy_prediction = dummy_model.predict(features_test)
mean_absolute_error(target_test, dummy_prediction)

8.04302925989673


# стандартизация и маштабирование
preprocessor_steps_linear = [('scaler', StandardScaler())]
preprocessor_linear = Pipeline(preprocessor_steps_linear)


# шаги для pipeline в GridSearchCV для моделей с деревями
steps_tree = [('regressor', DecisionTreeRegressor(random_state=rs))]


# шаги для pipeline в GridSearchCV для линейных моделей
steps_linear = [('preprocessor', preprocessor_linear), ('regressor', LinearRegression(n_jobs=-1))]


# шаги для pipeline в GridSearchCV для XGBoost
steps_xgb = [('regressor', xgb.XGBRegressor(random_state=rs, objective='reg:squarederror'))]


# параметры для улучшения моделей в GridSearchCV для моделей с деревьями

params_tree = [{'regressor': [DecisionTreeRegressor(random_state=rs)]
                , 'regressor__max_depth': [2, 5, 10, 15, 20]
                , 'regressor__max_features': ['sqrt'] #default = 1
                }
               , {'regressor': [RandomForestRegressor(random_state=rs)]
                  , 'regressor__bootstrap': [False]  #default = True
                  , 'regressor__max_depth': [2, 5, 7, 10, 15]
                  , 'regressor__max_features': ['sqrt'] #default = 1
                  , 'regressor__n_estimators': [2, 5, 10, 25, 50, 75, 100] # default = 100
                  }
              ]


# параметры для улучшения моделей в GridSearchCV для линейных алгоритмов

params_linear = [{'regressor': [LinearRegression()]}
                 , {'regressor': [Ridge()]
                    , 'regressor__alpha': [0.1 ,0.2, 0.5, 0.7, 1] # default 1.0
                    , 'regressor__solver': ['auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']
                    , 'regressor__random_state': [None, rs]
                    }
                 , {'regressor': [Lasso()]
                    , 'regressor__alpha': [0.1, 0.2, 0.5, 0.7, 1] # default 1.0
                    , 'regressor__random_state': [None, rs]
                    }
                 , {'regressor': [ElasticNet()]
                    , 'regressor__alpha': [0.1, 0.2, 0.5, 0.7, 1] # default 1.0
                    , 'regressor__l1_ratio': [0, 0.25, 0.5, 0.75, 1] # default 0.5
                    , 'regressor__random_state': [None, rs]
                    }
                 ]


# параметры для улучшения моделей в GridSearchCV для модели XGBoost
# ВАЖНО: установили фукцию потерь квадрат разницы

params_xgb = [{'regressor': [xgb.XGBRegressor(random_state=rs, objective='reg:squarederror')]
               , 'regressor__max_depth': [5, 7, 10]
               , 'regressor__n_estimators': [10, 25, 50, 75]
               , 'regressor__learning_rate': [0.1, 0.5, 0.9] #default = 1               
               }
             ]


#на вход идут признаки, целевой признак, шаги по умолчанию для pipeline,
# параметр для отображения процесса работы
# на выход лучший обученный объект GridSearchCV
#
def pipeline_gridsearchcv(X, y, steps, params, scoring, verbose = 1):
    
    # формируем итоговый pipeline 
    pipe = Pipeline(steps)
 
    # собираем все вместе, запускаем поиск
    grid = GridSearchCV(pipe, param_grid=params
                        , cv=cv_folders
                        , n_jobs=-1
                        , scoring=scoring
                        , verbose=verbose
                        , error_score='raise')
    grid.fit(X, y)
    
    #Посмотрим наилучшие подобранные параметры
    display('')
    display('Параметры лучшей модели:', grid.best_params_)
    display('Значение лучшей метрики качества:', -grid.best_score_)
    
    return grid


# создадим pipeline, подберем параметры для лучшей модели на кросс-валидации
grid_search_linear = pipeline_gridsearchcv(features_train
                                                   , target_train
                                                   , steps_linear
                                                   , params_linear
                                                   , 'neg_mean_absolute_error'
                                                   , verbose = 1)

Fitting 5 folds for each of 131 candidates, totalling 655 fits

''

'Параметры лучшей модели:'

{'regressor': Lasso(alpha=0.1),
 'regressor__alpha': 0.1,
 'regressor__random_state': None}

'Значение лучшей метрики качества:'

6.075135037621081


(pd.DataFrame(grid_search_linear.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


# создадим pipeline, подберем параметры для лучшей модели на кросс-валидации
grid_search_tree = pipeline_gridsearchcv(features_train
                                                   , target_train
                                                   , steps_tree
                                                   , params_tree
                                                   , 'neg_mean_absolute_error'
                                                   , verbose = 1)

Fitting 5 folds for each of 40 candidates, totalling 200 fits

''

'Параметры лучшей модели:'

{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt',
                       random_state=15823),
 'regressor__bootstrap': False,
 'regressor__max_depth': 15,
 'regressor__max_features': 'sqrt',
 'regressor__n_estimators': 100}

'Значение лучшей метрики качества:'

6.392086854375014


(pd.DataFrame(grid_search_tree.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


# создадим pipeline, подберем параметры для лучшей модели на кросс-валидации
grid_search_xgb = pipeline_gridsearchcv(features_train
                                        , target_train
                                        , steps_xgb
                                        , params_xgb
                                        , 'neg_mean_absolute_error'
                                        , verbose = 1)

Fitting 5 folds for each of 36 candidates, totalling 180 fits

''

'Параметры лучшей модели:'

{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
              interaction_constraints=None, learning_rate=0.1, max_bin=None,
              max_cat_threshold=None, max_cat_to_onehot=None,
              max_delta_step=None, max_depth=7, max_leaves=None,
              min_child_weight=None, missing=nan, monotone_constraints=None,
              n_estimators=75, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=15823, ...),
 'regressor__learning_rate': 0.1,
 'regressor__max_depth': 7,
 'regressor__n_estimators': 75}

'Значение лучшей метрики качества:'

5.990228459034357


(pd.DataFrame(grid_search_xgb.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


if abs(grid_search_tree.best_score_) < abs(grid_search_linear.best_score_):
    if abs(grid_search_tree.best_score_) < abs(grid_search_xgb.best_score_):
        best_model = grid_search_tree.best_estimator_
    else:
        best_model= grid_search_xgb.best_estimator_
else:
    if abs(grid_search_linear.best_score_) < abs(grid_search_xgb.best_score_):
        best_model = grid_search_linear.best_estimator_
    else:
        best_model= grid_search_xgb.best_estimator_
    
best_model

Pipeline(steps=[('regressor',
                 XGBRegressor(base_score=None, booster=None, callbacks=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, early_stopping_rounds=None,
                              enable_categorical=False, eval_metric=None,
                              feature_types=None, gamma=None, gpu_id=None,
                              grow_policy=None, importance_type=None,
                              interaction_constraints=None, learning_rate=0.1,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=7, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=75,
                              n_jobs=None, num_parallel_tree=None,
                              predictor=None, random_state=15823, ...))])


mean_absolute_error(target_test, best_model.predict(features_test))

5.670898227396192


best_model

Pipeline(steps=[('regressor',
                 XGBRegressor(base_score=None, booster=None, callbacks=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, early_stopping_rounds=None,
                              enable_categorical=False, eval_metric=None,
                              feature_types=None, gamma=None, gpu_id=None,
                              grow_policy=None, importance_type=None,
                              interaction_constraints=None, learning_rate=0.1,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=7, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=75,
                              n_jobs=None, num_parallel_tree=None,
                              predictor=None, random_state=15823, ...))])


best_model

Pipeline(steps=[('regressor',
                 XGBRegressor(base_score=None, booster=None, callbacks=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, early_stopping_rounds=None,
                              enable_categorical=False, eval_metric=None,
                              feature_types=None, gamma=None, gpu_id=None,
                              grow_policy=None, importance_type=None,
                              interaction_constraints=None, learning_rate=0.1,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=7, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=75,
                              n_jobs=None, num_parallel_tree=None,
                              predictor=None, random_state=15823, ...))])


data.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2324 entries, 1 to 2499
Data columns (total 31 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   bulk_1          2324 non-null   float64
 1   bulk_2          2324 non-null   float64
 2   bulk_3          2324 non-null   float64
 3   bulk_4          2324 non-null   float64
 4   bulk_5          2324 non-null   float64
 5   bulk_6          2324 non-null   float64
 6   bulk_7          2324 non-null   float64
 7   bulk_8          2324 non-null   float64
 8   bulk_9          2324 non-null   float64
 9   bulk_10         2324 non-null   float64
 10  bulk_11         2324 non-null   float64
 11  bulk_12         2324 non-null   float64
 12  bulk_13         2324 non-null   float64
 13  bulk_14         2324 non-null   float64
 14  bulk_15         2324 non-null   float64
 15  wire_1          2324 non-null   float64
 16  wire_2          2324 non-null   float64
 17  wire_3          2324 non-null   float64
 18  wire_4          2324 non-null   float64
 19  wire_5          2324 non-null   float64
 20  wire_6          2324 non-null   float64
 21  wire_7          2324 non-null   float64
 22  wire_8          2324 non-null   float64
 23  wire_9          2324 non-null   float64
 24  heating_time    2324 non-null   float64
 25  heating_cont    2324 non-null   int64  
 26  heating_energy  2324 non-null   float64
 27  total_time      2324 non-null   float64
 28  gaz             2324 non-null   float64
 29  start_temp      2324 non-null   float64
 30  finish_temp     2324 non-null   float64
dtypes: float64(30), int64(1)
memory usage: 581.0 KB


# обучающие признаки
display(features_train.shape, features_test.shape)
# целевой признак
display(target_train.shape, target_test.shape)

(1743, 30)

(581, 30)

(1743,)

(581,)


(pd.DataFrame(grid_search_linear.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


(pd.DataFrame(grid_search_tree.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


(pd.DataFrame(grid_search_xgb.cv_results_)
  [['params', 'mean_test_score', 'rank_test_score', 'mean_fit_time', 'mean_score_time']]
  .sort_values(by='mean_test_score', ascending=False)
  .head(10))


best_model

Pipeline(steps=[('regressor',
                 XGBRegressor(base_score=None, booster=None, callbacks=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, early_stopping_rounds=None,
                              enable_categorical=False, eval_metric=None,
                              feature_types=None, gamma=None, gpu_id=None,
                              grow_policy=None, importance_type=None,
                              interaction_constraints=None, learning_rate=0.1,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=7, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=75,
                              n_jobs=None, num_parallel_tree=None,
                              predictor=None, random_state=15823, ...))])


mean_absolute_error(target_test, best_model.predict(features_test))

5.670898227396192

	key	Начало нагрева дугой	Конец нагрева дугой	Активная мощность	Реактивная мощность
0	1	2019-05-03 11:02:14	2019-05-03 11:06:02	0.3051	0.2113
1	1	2019-05-03 11:07:28	2019-05-03 11:10:33	0.7657	0.4774
2	1	2019-05-03 11:11:44	2019-05-03 11:14:36	0.5803	0.4305
3	1	2019-05-03 11:18:14	2019-05-03 11:24:19	0.5185	0.3800
4	1	2019-05-03 11:26:09	2019-05-03 11:28:37	0.8671	0.6437

	key	Bulk 1	Bulk 2	Bulk 3	Bulk 4	Bulk 5	Bulk 6	Bulk 7	Bulk 8	Bulk 9	Bulk 10	Bulk 11	Bulk 12	Bulk 13	Bulk 14	Bulk 15
0	1	NaN	NaN	NaN	43.0000	NaN	NaN	NaN	NaN	NaN	NaN	NaN	206.0000	NaN	150.0000	154.0000
1	2	NaN	NaN	NaN	73.0000	NaN	NaN	NaN	NaN	NaN	NaN	NaN	206.0000	NaN	149.0000	154.0000
2	3	NaN	NaN	NaN	34.0000	NaN	NaN	NaN	NaN	NaN	NaN	NaN	205.0000	NaN	152.0000	153.0000
3	4	NaN	NaN	NaN	81.0000	NaN	NaN	NaN	NaN	NaN	NaN	NaN	207.0000	NaN	153.0000	154.0000
4	5	NaN	NaN	NaN	78.0000	NaN	NaN	NaN	NaN	NaN	NaN	NaN	203.0000	NaN	151.0000	152.0000

	key	Bulk 1	Bulk 2	Bulk 3	Bulk 4	Bulk 5	Bulk 6	Bulk 7	Bulk 8	Bulk 9	Bulk 10	Bulk 11	Bulk 12	Bulk 13	Bulk 14	Bulk 15
0	1	NaN	NaN	NaN	2019-05-03 11:28:48	NaN	NaN	NaN	NaN	NaN	NaN	NaN	2019-05-03 11:24:31	NaN	2019-05-03 11:14:50	2019-05-03 11:10:43
1	2	NaN	NaN	NaN	2019-05-03 11:36:50	NaN	NaN	NaN	NaN	NaN	NaN	NaN	2019-05-03 11:53:30	NaN	2019-05-03 11:48:37	2019-05-03 11:44:39
2	3	NaN	NaN	NaN	2019-05-03 12:32:39	NaN	NaN	NaN	NaN	NaN	NaN	NaN	2019-05-03 12:27:13	NaN	2019-05-03 12:21:01	2019-05-03 12:16:16
3	4	NaN	NaN	NaN	2019-05-03 12:43:22	NaN	NaN	NaN	NaN	NaN	NaN	NaN	2019-05-03 12:58:00	NaN	2019-05-03 12:51:11	2019-05-03 12:46:36
4	5	NaN	NaN	NaN	2019-05-03 13:30:47	NaN	NaN	NaN	NaN	NaN	NaN	NaN	2019-05-03 13:30:47	NaN	2019-05-03 13:34:12	2019-05-03 13:30:47

	key	Газ 1
0	1	29.7500
1	2	12.5556
2	3	28.5548
3	4	18.8412
4	5	5.4137

	key	Время замера	Температура
0	1	2019-05-03 11:02:04	1571.0000
1	1	2019-05-03 11:07:18	1604.0000
2	1	2019-05-03 11:11:34	1618.0000
3	1	2019-05-03 11:18:04	1601.0000
4	1	2019-05-03 11:25:59	1606.0000

	key	Wire 1	Wire 2	Wire 3	Wire 4	Wire 5	Wire 6	Wire 7	Wire 8	Wire 9
0	1	60.0600	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	2	96.0523	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	3	91.1602	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	4	89.0635	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	5	89.2382	9.1146	NaN	NaN	NaN	NaN	NaN	NaN	NaN

	key	Wire 1	Wire 2	Wire 3	Wire 4	Wire 5	Wire 6	Wire 7	Wire 8	Wire 9
0	1	2019-05-03 11:06:19	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	2	2019-05-03 11:36:50	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	3	2019-05-03 12:11:46	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	4	2019-05-03 12:43:22	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	5	2019-05-03 13:20:44	2019-05-03 13:15:34	NaN	NaN	NaN	NaN	NaN	NaN	NaN

	key	d_time	heating_energy		total_time
		sum	count	sum
0	1	1098.0000	5	628.6169	1583.0000
1	2	811.0000	4	395.2818	1144.0000
2	3	655.0000	5	581.7746	1525.0000
3	4	741.0000	4	543.7103	1093.0000
4	5	869.0000	4	412.1805	1362.0000

	key	start_temp	finish_temp
0	1	1571.0000	1613.0000
1	2	1581.0000	1602.0000
2	3	1596.0000	1599.0000
3	4	1601.0000	1625.0000
4	5	1576.0000	1602.0000

	bulk_1	bulk_2	bulk_3	bulk_4	bulk_5	bulk_6	bulk_7	bulk_8	bulk_9	bulk_10	...	wire_7	wire_8	wire_9	heating_time	heating_cont	heating_energy	total_time	gaz	start_temp	finish_temp
key
1	0.0000	0.0000	0.0000	43.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	...	0.0000	0.0000	0.0000	1098.0000	5	628.6169	1583.0000	29.7500	1571.0000	1613.0000
2	0.0000	0.0000	0.0000	73.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	...	0.0000	0.0000	0.0000	811.0000	4	395.2818	1144.0000	12.5556	1581.0000	1602.0000
3	0.0000	0.0000	0.0000	34.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	...	0.0000	0.0000	0.0000	655.0000	5	581.7746	1525.0000	28.5548	1596.0000	1599.0000
4	0.0000	0.0000	0.0000	81.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	...	0.0000	0.0000	0.0000	741.0000	4	543.7103	1093.0000	18.8412	1601.0000	1625.0000
5	0.0000	0.0000	0.0000	78.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	...	0.0000	0.0000	0.0000	869.0000	4	412.1805	1362.0000	5.4137	1576.0000	1602.0000

	params	mean_test_score	rank_test_score	mean_fit_time	mean_score_time
72	{'regressor': Lasso(alpha=0.1), 'regressor__alpha': 0.1, 'regressor__random_state': 15823}	-6.0751	1	0.0343	0.0133
90	{'regressor': ElasticNet(), 'regressor__alpha': 0.1, 'regressor__l1_ratio': 1, 'regressor__random_state': 15823}	-6.0751	1	0.0404	0.0117
89	{'regressor': ElasticNet(), 'regressor__alpha': 0.1, 'regressor__l1_ratio': 1, 'regressor__random_state': None}	-6.0751	1	0.0412	0.0097
71	{'regressor': Lasso(alpha=0.1), 'regressor__alpha': 0.1, 'regressor__random_state': None}	-6.0751	1	0.0311	0.0141
100	{'regressor': ElasticNet(), 'regressor__alpha': 0.2, 'regressor__l1_ratio': 1, 'regressor__random_state': 15823}	-6.0899	5	0.0509	0.0141
99	{'regressor': ElasticNet(), 'regressor__alpha': 0.2, 'regressor__l1_ratio': 1, 'regressor__random_state': None}	-6.0899	5	0.0420	0.0165
74	{'regressor': Lasso(alpha=0.1), 'regressor__alpha': 0.2, 'regressor__random_state': 15823}	-6.0899	5	0.0319	0.0101
73	{'regressor': Lasso(alpha=0.1), 'regressor__alpha': 0.2, 'regressor__random_state': None}	-6.0899	5	0.0275	0.0117
22	{'regressor': Ridge(), 'regressor__alpha': 0.2, 'regressor__random_state': 15823, 'regressor__solver': 'auto'}	-6.1055	9	0.0407	0.0226
24	{'regressor': Ridge(), 'regressor__alpha': 0.2, 'regressor__random_state': 15823, 'regressor__solver': 'cholesky'}	-6.1055	9	0.0331	0.0141

	params	mean_test_score	rank_test_score	mean_fit_time	mean_score_time
39	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 15, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 100}	-6.3921	1	1.4616	0.0643
38	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 15, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 75}	-6.4077	2	1.2109	0.0666
37	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 15, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 50}	-6.4478	3	0.8352	0.0335
36	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 15, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 25}	-6.4952	4	0.6244	0.0302
32	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 10, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 100}	-6.5041	5	1.2634	0.0686
31	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 10, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 75}	-6.5130	6	0.9116	0.0468
30	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 10, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 50}	-6.5209	7	0.6110	0.0383
29	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 10, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 25}	-6.5733	8	0.3245	0.0278
28	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 10, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 10}	-6.5778	9	0.1130	0.0157
35	{'regressor': RandomForestRegressor(bootstrap=False, max_depth=15, max_features='sqrt', random_state=15823), 'regressor__bootstrap': False, 'regressor__max_depth': 15, 'regressor__max_features': 'sqrt', 'regressor__n_estimators': 10}	-6.6541	10	0.2415	0.0351

	params	mean_test_score	rank_test_score	mean_fit_time	mean_score_time
7	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.1, 'regressor__max_depth': 7, 'regressor__n_estimators': 75}	-5.9902	1	1.8695	0.0302
3	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.1, 'regressor__max_depth': 5, 'regressor__n_estimators': 75}	-6.0183	2	1.6786	0.0662
11	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.1, 'regressor__max_depth': 10, 'regressor__n_estimators': 75}	-6.0579	3	1.6320	0.0193
13	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 5, 'regressor__n_estimators': 25}	-6.4765	4	0.3345	0.0109
16	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 7, 'regressor__n_estimators': 10}	-6.5278	5	0.1965	0.0194
12	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 5, 'regressor__n_estimators': 10}	-6.5295	6	0.1529	0.0125
17	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 7, 'regressor__n_estimators': 25}	-6.5697	7	0.6598	0.0242
14	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 5, 'regressor__n_estimators': 50}	-6.6002	8	0.7018	0.0165
15	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 5, 'regressor__n_estimators': 75}	-6.6779	9	1.1964	0.0186
18	{'regressor': XGBRegressor(base_score=None, booster=None, callbacks=None, colsample_bylevel=None, colsample_bynode=None, colsample_bytree=None, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=None, gpu_id=None, grow_policy=None, importance_type=None, interaction_constraints=None, learning_rate=0.1, max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None, max_delta_step=None, max_depth=7, max_leaves=None, min_child_weight=None, missing=nan, monotone_constraints=None, n_estimators=75, n_jobs=None, num_parallel_tree=None, predictor=None, random_state=15823, ...), 'regressor__learning_rate': 0.5, 'regressor__max_depth': 7, 'regressor__n_estimators': 50}	-6.6900	10	1.1939	0.0137

	key	wire_1	wire_2	wire_3	wire_4	wire_5	wire_6	wire_7	wire_8	wire_9
0	1	2019-05-03 11:06:19	NaT	NaT	NaT	NaT	NaT	NaT	NaT	NaT
1	2	2019-05-03 11:36:50	NaT	NaT	NaT	NaT	NaT	NaT	NaT	NaT
2	3	2019-05-03 12:11:46	NaT	NaT	NaT	NaT	NaT	NaT	NaT	NaT
3	4	2019-05-03 12:43:22	NaT	NaT	NaT	NaT	NaT	NaT	NaT	NaT
4	5	2019-05-03 13:20:44	2019-05-03 13:15:34	NaT	NaT	NaT	NaT	NaT	NaT	NaT

Проект: промышленность

Введение¶

Первичное описание проекта: промышленность¶

Библиотеки и настройки¶

Библиотеки¶

Настройки¶

Подготовка¶

Загрузка данных¶

Функция для загрузки¶

Информация об электродах¶

Информация о подаче сыпучих материалов: объём¶

Информация по подаче сыпучих материалов: время¶

Информация по продувке инертным газом¶

Информация об измерении температуры¶

Информация о проволочных материалах: объём¶

Информация о проволочных материалах: время¶

Выводы¶

Название колонок¶

Функция для названия колонок¶

Информация об электродах¶

Информация о подаче сыпучих материалов: объём¶

Информация о подаче сыпучих материалов: время¶

Информация по продувку инертным газом¶

Информация об измерении температуры¶

Информация о проволочных материалах: объём¶

Информация о проволочных материалах: время¶

Выводы¶

Тип колонок¶

Функция для даты¶

Информация об электродах¶

Информация по подаче сыпучих материалов: время¶

Информация об измерении температуры¶

Информация о проволочных материалах: время¶

Вывод¶

Исследовательский анализ данных, борьба с аномалиями¶

Информация об электродах¶

Выводы¶

Аномалии¶

Вывод¶

Анализ Реактивной мощности¶

Выводы¶

Информация по подаче сыпучих материалов: объём¶

Выводы¶

Заполнение пропусков¶

Вывод¶

Информация о подаче сыпучих материлов: время¶

Выводы¶

Информация по продувке инертным газом¶

Выводы¶

Информация об измерении температуры¶

Выводы¶

Аномалии¶

Выводы¶

Информация о проволочных материалах: объём¶

Выводы¶

Заполнение пропусков¶

Вывод¶

Информация о проволочных материалах: время¶

Выводы¶

Выводы¶

Результаты первой части¶

Список вопросов до уточняющей встречи¶

Цель проекта или формализация задачи.¶

Примерный план решения задачи¶

Объединение датафреймов: подготовка признаков для обучения¶

Подготовка датафреймов к объединению и дополнительные признаки¶

Информация об электродах¶

Информация об измерении температуры¶

Объединение датафреймов.¶

Познакомимся с нашими объединенными данными¶

Выводы¶

Заполнение пропусков¶

Номер партии - новый индекс¶

Выводы¶

Обучение моделей¶

Генератор псевдослучайных значение¶

Метрика и функция потерь¶

Целевой и обучающие признаки¶

Разбиение данных на выборки¶

Кросс-валидация¶